REINFORCEMENT LEARNING AND OPTIMAL CONTROL METHODS FOR UNCERTAIN NONLINEAR SYSTEMS By SHUBHENDU BHASIN A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy REINFORCEMENT LEARNING AND OPTIMAL CONTROL METHODS FOR UNCERTAIN NONLINEAR SYSTEMS By Shubhendu Bhasin August 2011 Chair: Warren E. Dixon Major: Mechanical Engineering Notions of optimal behavior expressed in natural systems led researchers to develop reinforcement learning (RL) as a computational tool in machine learning to learn actions by trial and error interactions yielding either a reward or punishment. RL provides a way for learning agents to optimally interact with uncertain complex environments, and hence, can address problems from a variety of domains, including artificial intelligence, controls, economics, operations research, etc. The focus of this work is to investigate the use of RL methods in feedback control to improve the closed-loop performance of nonlinear systems. Most RL-based controllers are limited to discrete-time systems, are offline methods, require knowledge of system dynamics and/or lack a rigorous stability analysis. This research investigates new control methods as an approach to address some of the limitations associated with traditional RL-based controllers. A robust adaptive controller with an adaptive critic or actor-critic (AC) architecture is developed for a class of uncertain nonlinear systems with disturbances. The AC structure is inspired from RL and uses a two pronged neural network (NN) architecture – an action NN, also called the actor, which approximates the plant dynamics and generates appropriate control actions; and a critic NN, which evaluates the performance of the actor, based on some performance index.